The present invention relates generally to speech recognition systems and, more particularly, to methods and apparatus for providing fast speaker adaptation of acoustic models used in these systems.
The performance of a general speaker-independent (SI) system is often poor for specific tasks in automatic speech recognition (ASR). Speaker adaptation techniques have recently emerged as both effective and practical methods to improve the performance of an ASR. For instance, for telephony tasks, adaptation could improve the performance significantly by compensating for the uncertainty and other effects of the telephone channel.
The most common techniques for adaptation change the acoustic models used in speech recognition using samples of speech data from a particular speaker or environment. These acoustic models attempt to model the probability density function (pdf) of acoustic feature vectors for different sounds (phones). It is common to use parametric pdf""s such as gaussians to approximate the true pdf of the data. These gaussians are parametrized by means (xcexci, i=1, . . . , N) and covariances (xcexa3i, i=1, . . . , N) in high dimensional (D-dimensional) feature space.
One common technique to adapt the gaussians is maximum likelihood linear regression (MLLR) adaptation. MLLR adaptation is described in C. J. Legetter and P. C. Woodland, xe2x80x9cMaximum Likelihood Linear Regression for Speaker Adaptation of Continuous density HMM""s,xe2x80x9d Computer Speech and Language, vol. 9, no. 2, pp. 171-186, 1995, the disclosure of which is incorporated herein by reference. In this technique, the adapted gaussians that better model the speakers data are assumed to be derivable from the speaker independent gaussians by the application of an affine transform. Consequently, if xcexci represents one of the speaker independent gaussians, the speaker-adapted gaussian is assumed to be {circumflex over (xcexc)}i=Axcexci where A is a Dxc3x97D dimensional matrix and xcexci is a Dxc3x971 dimensional vector.
Another common technique that is used in speaker adaptation is maximum a posteriori (MAP) adaptation. MAP adaptation is described in J. L. Gauvain and C. H. Lee, xe2x80x9cMaximum-a-Posteriori estimation for multivariate gaussian observations of Markov chains,xe2x80x9d IEEE Trans. Speech and Audio Processing, vol. 2, no. 2, pp. 291-298, April 1994, the disclosure of which is incorporated herein by reference. Here, the features vectors in the adaptation data are assigned (with some probability) to the gaussians of the speaker independent system, and based on this assignment, the zero-th, first and second order statistics are accumulated for each gaussian. These accumulated statistics are then smoothed back with the sufficient statistics of the gaussians as computed from the training data (from which the gaussians were estimated), and the smoothed statistics are then converted into the means and covariances of the adapted gaussians.
For telephony applications, adaptation raises two concerns. First, typically, there is only as little as two seconds of data from which to adapt. This makes it imperative, especially for real-time applications, that the adaptation be fast. Second, with so little data, many of the parameters required for MLLR or MAP cannot be robustly estimated.
Since most of the telephony tasks mandate real-time processing, the SI system has to be implemented in a way that allows fast computation of gaussian likelihood values. It is to be understood that a system usually has tens of thousands of gaussians. The full computation associated with these gaussians is prohibitively expensive. One common approach to speed up the computation is to organize the gaussians in a hierarchical fashion so that only a subset of the gaussians need to be computed at any time. This is called hierarchical clustering of gaussians. Furthermore, the parameter space (means and covariances) of the gaussians is quantized. The gaussians model probability density functions in a D-dimensional feature space. These dimensions are divided into subsets of dimensions (called bands) and the gaussian""s parameters (means, variances and priors) in each band are vector quantized. This is referred to as a band-quantized (BQ) system. Typically, D is 40 dimensional, there are 20 bands with 2 dimensions in each band, and the gaussians of the original system in each band are quantized into 128 bins (referred to as atoms, hence atoms represent quantized gaussians in two dimensions). The original gaussians are now represented by the closest atom in each band.
The process of computing the likelihood of a feature vector for a given gaussian is now as follows. For each band, the likelihood of the feature vector values in the band is computed for each atom in that band (this represents the computation of 128*20 2-dimensional gaussian likelihoods). Subsequently, the likelihood for the original gaussian is obtained by looking up the likelihood values of the atoms associated with the gaussian. Hence, the BQ system stores each gaussian as an index mapping (array of indices) to atoms, and also the hierarchy of mappings in the hierarchical clustering.
By way of example, the standard procedure for MLLR adaptation with the BQ system is as follows. The means (and possibly covariances) of the Dimensional gaussians are transformed into a new set of gaussians. Subsequently, based on acoustic similarity, these gaussians are clustered into hierarchical groups. Finally, for each band, the new gaussians are vector quantized to form a set of atoms. Unfortunately, this process cannot meet the real-time requirements of various applications such as, for example, telephony applications.
The present invention provides for methods and apparatus for improved speaker model adaptation with application to a speech recognition system that uses band quantized gaussians for the acoustic model. In most techniques, the speaker adapted gaussians are derived from the complete un-quantized speaker independent gaussians by some technique, and the speaker adapted gaussians are then re-quantized. The mapping of speaker independent gaussians to speaker adapted gaussians may be through an affine transformation (MLLR) or through other technique (such as the smoothing of counts, as in MAP adaptation). The present invention is applicable to all these techniques.
In the present invention, we impose certain constraints on the mapping function that are related to the division of dimensions into bands. These constraints enable the mapping function to be computed on the basis of the un-quantized gaussians, but then enable the mapping to be applied directly onto the atoms of the quantized speaker independent system, hence eliminating the overhead of having to re-quantize the adapted gaussians.
Advantageously, in accordance with the invention, by applying the adaptation directly to the atoms, the hierarchy and index mappings are left unchanged. When the mapping is properly designed with respect to the partitioning of the original feature space into bands, the computation of the transform becomes easy and hence the adaptation is fast. Additionally, since the number of adapted parameters is fairly small, the process is also more robust.
In one embodiment of the invention, the mapping function is assumed to be an affine transform, i.e., the parameters of the adapted gaussians are related to the speaker independent parameters by means of an affine transform. The application of our invention to this mapping function constrains the transformation to be block diagonal with the blocks corresponding to the dimension-bands in the system. The transforms are computed in order to maximize the likelihood of the adaptation data, or alternately some other objective function, e.g., a Bayesian likelihood, discrimination, etc., may also be used to compute the transform in accordance with the invention. It is to be understood that the adaptation data may be a short decoded sequence of speech from the speaker for whom the acoustic models are being adapted.
In another embodiment of the invention, the mapping function is assumed to maximize the Bayesian likelihood of the adaptation data (i.e., the probability of the class conditioned on the acoustic observations). This adaptation process requires the assignment (with some probability) of each feature vector in the adaptation data to a class, and the accumulation of the first and second order statistics of the feature vectors for each atom. The probability of assigning a feature vector to a class is computed using the quantized speaker independent gaussians. The sufficient statistics collected from the adaptation data for each atom are smoothed back to the sufficient statistics of the speaker independent atoms, and the smoothed statistics are then converted into gaussians which represent the new atoms.
The real-time adaptation techniques of the invention maybe employed in any suitable automatic speech recognition system for use in a variety of applications such as, for example but not limited to, telephony applications.
These and other objects, features and advantages of the present invention will become apparent from the following detailed description of illustrative embodiments thereof, which is to be read in connection with the accompanying drawings.